Skip to main content Skip to complementary content

Setting general connection properties

This section describes how to configure general connection properties. For an explanation of how to configure advanced connection properties, see Setting advanced connection properties.

To add an Amazon S3 target endpoint to Qlik Replicate:

  1. In Tasks view, click Manage Endpoint Connections to open the Manage Endpoints Connections dialog box. Then click the New Endpoint Connection button. For more information on adding an endpoint to Qlik Replicate, see Defining and managing endpoints.
  2. In the Name field, type a name for your endpoint. This can be any name that will help to identify the endpoint being used.
  3. Optionally, in the Description field, type a description that helps to identify the endpoint.
  4. Select Target as the endpoint role.
  5. Select Amazon S3 as the endpoint Type.
  6. Configure the remaining settings in the General tab as described below.

    Amazon S3 Storage

    This section describes the Amazon S3 storage options.

    Bucket name

    The name of your Amazon S3 bucket.

    Bucket region

    The region where your bucket is located. It is recommended to leave the default (Auto-Detect) as it usually eliminates the need to select a specific region. However, due to security considerations, for some regions (for example, AWS GovCloud) you might need to explicitly select the region. If the region you require does not appear in the regions list, select Other and specify the code in the Region code field.

    For a list of region codes, see AWS Regions.

    Use AWS PrivateLink

    Select this to connect to an Amazon VPC and then specify the VPC Endpoint URL (for example, https://bucket.vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com).

    Access options

    Choose one of the following:

    • Key pair

      Choose this method to authenticate with your Access Key and Secret Key.

    • IAM Roles for EC2

      Choose this method if the machine on which Qlik Replicate is installed is configured to authenticate itself using an IAM role.

      For information on IAM roles, see IAM roles.

    • Security Token Service (STS)

      Choose this method to authenticate using SAML 2.0 with Active Directory Federation Services.

      For more information about this access option, see About SAML 2.0-based federation.

    • IAM Roles Anywhere

      IAM Roles Anywhere can be set up in the IAM Roles Anywhere console, via the AWS CLI, or using the AWS SDK. IAM Roles Anywhere allows you to use your private key infrastructure (PKI) to generate temporary credentials for accessing IAM roles from outside of AWS. This means you can securely access AWS resources from Replicate without having to manage long-term credentials.

      When this option is selected, specify the following:

      • Certificate file: Path to the Replicate public certificate in PEM format. This file needs to be signed with the CA certificate configured in the IAM Roles Anywhere console.
      • Private key file: Path to the Replicate private key file in PEM format.
      • Private key passphrase: The private key passphrase. Only required if the private key file is encrypted.
      • Trust anchor ARN: The ARN associated with the trust anchor you created in the IAM Roles Anywhere console. You establish trust between IAM Roles Anywhere and your certificate authority (CA) by creating a trust anchor. A trust anchor is a reference to either AWS Private CA or an external CA certificate. Your workloads outside of AWS authenticate with the trust anchor using certificates issued by the trusted CA in exchange for temporary AWS credentials.
      • Profile ARN: The ARN associated with the profile you created in the IAM Roles Anywhere console. To specify which roles IAM Roles Anywhere assumes and what your workloads can do with the temporary credentials, you create a profile. In a profile, you can define permissions with IAM managed policies to limit the permissions for a created session.
      • Role ARN: The ARN associated with the role you created in the IAM Roles Anywhere console. A role is an IAM identity that you create in your account with specific permissions. For IAM Roles Anywhere to be able to assume a role and deliver temporary AWS credentials, the role must trust the IAM Roles Anywhere service principal.

      For more information about IAM Roles Anywhere, see:

      Extend AWS IAM roles to workloads outside of AWS with IAM Roles Anywhere

    Access key

    The access key information for Amazon S3.

    Information noteThis option is only available when Key pair is the access option.

    Secret key

    The secret key information for Amazon S3.

    Information noteThis option is only available when Key pair is the access option.

    ADFS URL

    The URL to an Active Directory Federation Services page, responsible for returning a SAML claims document to be sent over to AWS.

    Information noteThis option is only available when Security Token Service (STS) is the access option.

    AD principal name

    The principal (user) name to use when identifying against ADFS

    The format should be: user.name@domain

    Information noteThis option is only available when Security Token Service (STS) is the access option.

    AD principal password

    The principal password to use when identifying against ADFS

    IdP ARN

    The Amazon Resource Name (ARN) of the Active Directory issuing the SAML claims document. This is required as it enables AWS to identify the signer of the SAML document and verify its signature.

    SAML Role ARN

    The Amazon Resource Name (ARN) of the specific role the returned credentials should be assigned.

    Switch role after assuming SAML role

    Use this option to switch role after authentication.

    For more information, see:

    https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_aws-accounts.html

    When this option is selected, the following information is required:

    • Role ARN

      The ARN associated with the target role.

    • Role external ID

      The value of the external ID condition in the target role’s trust policy.

    Target folder

    The target folder in your Amazon S3 bucket.

    File Attributes

    Delimiters can be standard characters or a hexadecimal (hex) value. Note that the "0x" prefix must be used to denote a hexadecimal delimiter (e.g. 0x01 = SOH). In the Field delimiter, Record delimiter and Null value fields, the delimiter can consist of concatenated hex values (e.g. 0x0102 = SOHSTX), whereas in the Quote character and Escape character fields, it can only be a single hex value.

    The hexadecimal number 0x00 is not supported (i.e. only 0x01-0xFF are supported).

    Format

    You can choose to create the target files in CSV or JSON, or Parquet format.

    In a JSON file, each record is represented by a single line, as in the following example:

    { "book_id": 123, "title": "Alice in Wonderland", "price": 6.99, "is_hardcover": false }

    { "book_id": 456, "title": "Winnie the Pooh", "price": 6.49, "is_hardcover": true }

    { "book_id": 789, "title": "The Cat in the Hat", "price": 7.23, "is_hardcover": true }

    See also: Content-type and content-encoding properties.

    Information noteChanging the format (for example, from CSV to JSON or from JSON to CSV) while the task is in a stopped state and then resuming the task, is not supported.
    Information noteIf you choose JSON or Parquet format , the following fields will be hidden as they are only relevant to CSV format: Field delimiter, Record delimiter, Null value, Quote character, Escape character, and Add metadata header.

    For information about data type mappings when using Parquet format and limitations, see Mapping from Qlik Replicate data types to Parquet and Limitations and considerations.

    Field delimiter

    The delimiter that will be used to separate fields (columns) in the target files. The default is a comma.

    Example using a comma as a delimiter:

    "mike","male"

    Record delimiter

    The delimiter that will be used to separate records (rows) in the target files. The default is a newline (\n).

    Example:

    "mike","male"\n
    "sara","female"\n

    Null value

    The string that will be used to indicate a null value in the target files.

    Example (where \n is the record delimiter and @ is the null value):

    "mike","male",295678\n
    "sara","female",@\n

    Quote character

    The character that will be used at the beginning and end of a text column. The default is the double-quote character ("). When a column that contains column delimiters is enclosed in double-quotes, the column delimiter characters are interpreted as actual data, and not as column delimiters.

    Example (where a @ is the quote character):

    @mike@,@male@

    Quote escape character

    The character used to escape a quote character in the actual data. The default is the double-quote character (").

    Example (where " is the quote character and \ is the escape character):

    1955,"old, \"rare\", Chevrolet","$1000"

    Add metadata header

    You can optionally add a header row to the data files. The header row can contain the source column names and/or the intermediate (i.e. Replicate) data types.

    Example of a target file with a header row when both With column names and With data types are selected:

    Position:DECIMAL(38,0),Color:VARCHAR(10)

    1,"BLUE"

    2,"BROWN"

    3,"RED"

    ...

    Maximum file size

    The maximum size a file can reach before it is closed (and optionally compressed). This value applies both to data files and to Reference Files.

    For information on generating reference files, see Setting advanced connection properties.

    Compress files using

    Choose one of the compression options to compress the target files or NONE (the default) to leave them uncompressed. Note that the available compressions options are determined by the selected file format.

    Change Processing

    This section describes conditional settings in Change Processing.

    Apply/Store changes when

    • File size reaches: Specify the maximum size of Change Data to accumulate before uploading the file to Amazon S3.
    • Elapsed time reaches: Elapsed time reaches x.

    Metadata files

    This section describes the Metadata files option.

    Create metadata files in the target folder

    When this option is selected, for each data file, a matching metadata file with a .dfm extension will be created under the specified target folder. The metadata files provide additional information about the task/data such as the source endpoint type, the source table name, the number of records in the data file, and so on.

    For a full description of the metadata file as well as possible uses, see Metadata file description .

    Data encryption

    This section describes the Data encryption options.

    Choose one of the following:

  7. To determine if the connection information you entered is correct, click Test Connection. If the connection test is successful, click Save.
Information note

As part of connection testing process, Replicate uploads a test file to the specified Amazon S3 Target folder and then deletes it once a connection has been established.

If the connection is successful a message in green is displayed. If the connection fails, an error message is displayed at the bottom of the dialog box.

To view the log entry if the connection fails, click View Log. The server log is displayed with the information for the connection failure. Note that this button is not available unless the test connection fails.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!